So far every pathway analysis method we’ve covered relies on some information about groups of samples in our data. For over-representation analysis (ORA), we created gene sets from two different two group comparisons. In the Gene Set Enrichment Analysis (GSEA) example, we used statistics from a differential gene expression (DGE) analysis where we compared MYCN amplified cell lines to non-amplified cell lines; we needed that amplification status information.

What if we’re less sure about groups in our data or we want to analyze our data in a more unsupervised manner?

In this notebook we will cover a method called Gene Set Variation Analysis (GSVA) (Hänzelmann, Castelo, and Guinney. 2013) that allows us to calculate gene set or pathway scores on a per-sample basis.

We like this quote from the GSVA paper (Hänzelmann et al. 2013) to set the stage:

While [gene set enrichment] methods are generally regarded as end points of a bioinformatic analysis, GSVA constitutes a starting point to build pathway-centric models of biology.

Rather than contextualizing some results you already have from another analysis like DGE, GSVA is designed to provide an estimate of pathway variation for each of the samples in an experiment. Note that these scores will depend on the samples included in the dataset when you run GSVA; if you added more samples and reran GSVA, you would expect the scores to change.

Set up

Libraries

# Pipes
library(magrittr)
# Gene Set Variation Analysis
library(GSVA)

Directories and files

Directories

# We have some medulloblastoma data from the OpenPBTA project that we've 
# prepared ahead of time
input_dir <- file.path("data", "open-pbta")

# Create a directory specifically for the results using this dataset
output_dir <- file.path("results", "open-pbta") 
if (!dir.exists(output_dir)) {
  dir.create(output_dir, recursive = TRUE)
}

Input

We have VST transformed RNA-seq data, annotated with gene symbols, that has been collapsed such that there are no duplicated gene identifiers (see setup).

rnaseq_file <- file.path(input_dir, "medulloblastoma_vst_collapsed.tsv")

Output

gsva_results_file <- file.path(output_dir, "medulloblastoma_gsva_results.tsv")

Gene sets

The function that we will use to run GSVA wants the gene sets to be in a list, rather than a tidy data frame that we used with clusterProfiler (although it does accept other formats).

We’re going to take this opportunity to introduce a different format that gene sets are often distributed in called GMT (Gene Matrix Transposed).

We’re going to read in the Hallmark collection file directly from MSigDB, rather than using msigdbr like we did in earlier notebooks.

The RNA-seq data uses gene symbols, so we need gene sets that use gene symbols, too.

# R can often read in data from a URL
hallmarks_url <- "https://data.broadinstitute.org/gsea-msigdb/msigdb/release/7.1/h.all.v7.1.symbols.gmt"

# QuSAGE is another pathway analysis method, the qusage package has a function
# for reading GMT files and turning them into a list
hallmarks_list <- qusage::read.gmt(hallmarks_url)

What does this list look like?

head(hallmarks_list)

RNA-seq data

We have VST transformed RNA-seq data, which is on a log2-like scale. These data are from the Open Pediatric Brain Tumor Atlas (OpenPBTA) OpenPBTA is a collaborative project organized by the CCDL and the Center for Data-Driven Discovery in Biomedicine (D3b) at the Children’s Hospital of Philadelphia conducted openly on GitHub.

You can read more about the project here.

We’re only working with the medulloblastoma samples in this example.

rnaseq_df <- readr::read_tsv(rnaseq_file)

── Column specification ──────────────────────────────────────────────────────────────────────────────────────────
cols(
  .default = col_double(),
  gene_symbol = col_character()
)
ℹ Use `spec()` for the full column specifications.
# What does the RNA-seq data frame look like?
rnaseq_df[1:5, 1:5]

For GSVA, we need a matrix.

rnaseq_mat <- rnaseq_df %>%
  tibble::column_to_rownames("gene_symbol") %>%
  as.matrix()

Note: If we had duplicate gene symbols here, we couldn’t set them as rownames.

GSVA

Figure 1 from Hänzelmann, Castelo, and Guinney. (2013).

You may notice that GSVA has some commonalities with GSEA. Rather than ranking genes based on some statistic we selected ahead of time, GSVA fits a model and ranks genes based on their expression level relative to the sample distribution. This is a way of asking if a gene i is highly or lowly expressed in a sample j in the context of this experiment and ranking accordingly (Hänzelmann et al. 2013). The pathway-level score calculated is a way of asking how genes within a gene set vary as compared to genes that are outside of that gene set (Malhotra. 2018). (This is sometimes called a competitive test in gene set enrichment literature.) The intuition here is that we will get pathway-level scores for each sample that indicate if genes in a pathway vary concordantly in one direction (overexpressed or underexpressed relative to the overall population) (Hänzelmann et al. 2013).

The output is a gene set by sample matrix of GSVA scores.

Perform GSVA

gsva_results <- gsva(rnaseq_mat, 
                     hallmarks_list,
                     method = "gsva",
                     # Appropriate for our transformed data on log2-like scale
                     kcdf = "Gaussian",  
                     # Minimum gene set size
                     min.sz = 15, 
                     # Maximum gene set size
                     max.sz = 500,
                     # Compute Gaussian-distributed scores
                     mx.diff = TRUE)
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 50 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                              
  |                                                                                                        |   0%
  |                                                                                                              
  |==                                                                                                      |   2%
  |                                                                                                              
  |====                                                                                                    |   4%
  |                                                                                                              
  |======                                                                                                  |   6%
  |                                                                                                              
  |========                                                                                                |   8%
  |                                                                                                              
  |==========                                                                                              |  10%
  |                                                                                                              
  |============                                                                                            |  12%
  |                                                                                                              
  |===============                                                                                         |  14%
  |                                                                                                              
  |=================                                                                                       |  16%
  |                                                                                                              
  |===================                                                                                     |  18%
  |                                                                                                              
  |=====================                                                                                   |  20%
  |                                                                                                              
  |=======================                                                                                 |  22%
  |                                                                                                              
  |=========================                                                                               |  24%
  |                                                                                                              
  |===========================                                                                             |  26%
  |                                                                                                              
  |=============================                                                                           |  28%
  |                                                                                                              
  |===============================                                                                         |  30%
  |                                                                                                              
  |=================================                                                                       |  32%
  |                                                                                                              
  |===================================                                                                     |  34%
  |                                                                                                              
  |=====================================                                                                   |  36%
  |                                                                                                              
  |========================================                                                                |  38%
  |                                                                                                              
  |==========================================                                                              |  40%
  |                                                                                                              
  |============================================                                                            |  42%
  |                                                                                                              
  |==============================================                                                          |  44%
  |                                                                                                              
  |================================================                                                        |  46%
  |                                                                                                              
  |==================================================                                                      |  48%
  |                                                                                                              
  |====================================================                                                    |  50%
  |                                                                                                              
  |======================================================                                                  |  52%
  |                                                                                                              
  |========================================================                                                |  54%
  |                                                                                                              
  |==========================================================                                              |  56%
  |                                                                                                              
  |============================================================                                            |  58%
  |                                                                                                              
  |==============================================================                                          |  60%
  |                                                                                                              
  |================================================================                                        |  62%
  |                                                                                                              
  |===================================================================                                     |  64%
  |                                                                                                              
  |=====================================================================                                   |  66%
  |                                                                                                              
  |=======================================================================                                 |  68%
  |                                                                                                              
  |=========================================================================                               |  70%
  |                                                                                                              
  |===========================================================================                             |  72%
  |                                                                                                              
  |=============================================================================                           |  74%
  |                                                                                                              
  |===============================================================================                         |  76%
  |                                                                                                              
  |=================================================================================                       |  78%
  |                                                                                                              
  |===================================================================================                     |  80%
  |                                                                                                              
  |=====================================================================================                   |  82%
  |                                                                                                              
  |=======================================================================================                 |  84%
  |                                                                                                              
  |=========================================================================================               |  86%
  |                                                                                                              
  |============================================================================================            |  88%
  |                                                                                                              
  |==============================================================================================          |  90%
  |                                                                                                              
  |================================================================================================        |  92%
  |                                                                                                              
  |==================================================================================================      |  94%
  |                                                                                                              
  |====================================================================================================    |  96%
  |                                                                                                              
  |======================================================================================================  |  98%
  |                                                                                                              
  |========================================================================================================| 100%

Note: the gsva() documentation says we can use kcdf = "Gaussian" if we had RNA-seq log-CPMs, log-RPKMs or log-TPMs, but we would use kcdf = "Poisson" on integer counts.

# Let's explore what the output of gsva() looks like
gsva_results[1:5, 1:5]
                                    BS_09Z7TC35 BS_1AYRM596 BS_1BWP5MCT BS_1QXEC43H BS_1TWCV047
HALLMARK_TNFA_SIGNALING_VIA_NFKB    -0.44911439   0.5208667  -0.5609193  0.32300630  -0.1468081
HALLMARK_HYPOXIA                    -0.38297104   0.2436910  -0.5058759  0.36247083  -0.2971559
HALLMARK_CHOLESTEROL_HOMEOSTASIS    -0.26534735   0.1054224  -0.5180933  0.32418657  -0.4561386
HALLMARK_MITOTIC_SPINDLE             0.12727006   0.2339489  -0.4338076 -0.07023068  -0.1861483
HALLMARK_WNT_BETA_CATENIN_SIGNALING -0.09287646   0.1602124  -0.4427594  0.41372523  -0.1152437

A note on gene set size

Often the scores of gene set enrichment methods are not comparable between gene sets of different sizes. Let’s do an experiment using randomly generated gene sets to explore this idea a bit more.

We need to get a collection of all possible genes we will sample from to create random gene sets. Because we’re doing some random sampling, we need to set a seed for this to be reproducible.

# Use all the gene symbols in the dataset as the pool of possible genes
all_genes <- rownames(rnaseq_mat)

# Set a seed for reproducibility
set.seed(2020)

Our minimum gene set size earlier was 15 genes and our maximum gene set size was 500 genes. We’ll use the same minimum and maximum values for our random gene sets and some values in between.

# Make a list of integers that indicate the random gene set sizes
gene_set_size <- list(15, 25, 50, 100, 250, 500)

For each gene set size, we will generate 100 random gene sets

# Set number of replicates
nreps <-  100
# Generate 100 random gene sets of each size
random_gene_sets <- 
  purrr::map(
    rep(gene_set_size, nreps),  # Repeat gene sizes so we run `nreps` times 
    # Sample the vector of all genes, choosing the number of items specified
    # in the element of gene set size
    ~ base::sample(x = all_genes,
                   size = .x)
  )

The Hallmarks list we used earlier stored the gene set names as the name of the list, so let’s add names to our random gene sets that indicate what size they are and so gsva() doesn’t get upset.

# We will include the size of the gene set in the gene set name 
# Start by taking the length of each pathway and appending "pathway_" to that
# number
lengths_vector <- purrr::map(random_gene_sets, ~ length(.x)) %>%
  purrr::map(~ paste0("pathway_", .x)) %>%
  # Return a vector
  purrr::flatten_chr()

# Add the names in lengths_vector to the list
random_gene_sets <- random_gene_sets %>% 
  # make.names() appends a "version" if something is not unique
  purrr::set_names(nm = make.names(lengths_vector, unique = TRUE))

Run GSVA on our dataset with the same parameters as before, but now with random gene sets.

random_gsva_results <- gsva(rnaseq_mat, 
                            random_gene_sets,
                            method = "gsva",
                            # Appropriate for our transformed data on 
                            # log2-like scale
                            kcdf = "Gaussian",  
                            # Minimum gene set size
                            min.sz = 15, 
                            # Maximum gene set size
                            max.sz = 500,
                            # Compute Gaussian-distributed scores
                            mx.diff = TRUE)                            
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 579 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                              
  |                                                                                                        |   0%
  |                                                                                                              
  |=                                                                                                       |   1%
  |                                                                                                              
  |==                                                                                                      |   2%
  |                                                                                                              
  |===                                                                                                     |   2%
  |                                                                                                              
  |===                                                                                                     |   3%
  |                                                                                                              
  |====                                                                                                    |   3%
  |                                                                                                              
  |====                                                                                                    |   4%
  |                                                                                                              
  |=====                                                                                                   |   4%
  |                                                                                                              
  |=====                                                                                                   |   5%
  |                                                                                                              
  |======                                                                                                  |   5%
  |                                                                                                              
  |======                                                                                                  |   6%
  |                                                                                                              
  |=======                                                                                                 |   6%
  |                                                                                                              
  |=======                                                                                                 |   7%
  |                                                                                                              
  |========                                                                                                |   7%
  |                                                                                                              
  |========                                                                                                |   8%
  |                                                                                                              
  |=========                                                                                               |   8%
  |                                                                                                              
  |=========                                                                                               |   9%
  |                                                                                                              
  |==========                                                                                              |   9%
  |                                                                                                              
  |==========                                                                                              |  10%
  |                                                                                                              
  |===========                                                                                             |  10%
  |                                                                                                              
  |===========                                                                                             |  11%
  |                                                                                                              
  |============                                                                                            |  11%
  |                                                                                                              
  |============                                                                                            |  12%
  |                                                                                                              
  |=============                                                                                           |  12%
  |                                                                                                              
  |=============                                                                                           |  13%
  |                                                                                                              
  |==============                                                                                          |  13%
  |                                                                                                              
  |==============                                                                                          |  14%
  |                                                                                                              
  |===============                                                                                         |  14%
  |                                                                                                              
  |===============                                                                                         |  15%
  |                                                                                                              
  |================                                                                                        |  15%
  |                                                                                                              
  |================                                                                                        |  16%
  |                                                                                                              
  |=================                                                                                       |  16%
  |                                                                                                              
  |=================                                                                                       |  17%
  |                                                                                                              
  |==================                                                                                      |  17%
  |                                                                                                              
  |==================                                                                                      |  18%
  |                                                                                                              
  |===================                                                                                     |  18%
  |                                                                                                              
  |===================                                                                                     |  19%
  |                                                                                                              
  |====================                                                                                    |  19%
  |                                                                                                              
  |====================                                                                                    |  20%
  |                                                                                                              
  |=====================                                                                                   |  20%
  |                                                                                                              
  |=====================                                                                                   |  21%
  |                                                                                                              
  |======================                                                                                  |  21%
  |                                                                                                              
  |======================                                                                                  |  22%
  |                                                                                                              
  |=======================                                                                                 |  22%
  |                                                                                                              
  |========================                                                                                |  23%
  |                                                                                                              
  |=========================                                                                               |  24%
  |                                                                                                              
  |==========================                                                                              |  25%
  |                                                                                                              
  |===========================                                                                             |  26%
  |                                                                                                              
  |============================                                                                            |  27%
  |                                                                                                              
  |=============================                                                                           |  27%
  |                                                                                                              
  |=============================                                                                           |  28%
  |                                                                                                              
  |==============================                                                                          |  28%
  |                                                                                                              
  |==============================                                                                          |  29%
  |                                                                                                              
  |===============================                                                                         |  29%
  |                                                                                                              
  |===============================                                                                         |  30%
  |                                                                                                              
  |================================                                                                        |  30%
  |                                                                                                              
  |================================                                                                        |  31%
  |                                                                                                              
  |=================================                                                                       |  31%
  |                                                                                                              
  |=================================                                                                       |  32%
  |                                                                                                              
  |==================================                                                                      |  32%
  |                                                                                                              
  |==================================                                                                      |  33%
  |                                                                                                              
  |===================================                                                                     |  33%
  |                                                                                                              
  |===================================                                                                     |  34%
  |                                                                                                              
  |====================================                                                                    |  34%
  |                                                                                                              
  |====================================                                                                    |  35%
  |                                                                                                              
  |=====================================                                                                   |  35%
  |                                                                                                              
  |=====================================                                                                   |  36%
  |                                                                                                              
  |======================================                                                                  |  36%
  |                                                                                                              
  |======================================                                                                  |  37%
  |                                                                                                              
  |=======================================                                                                 |  37%
  |                                                                                                              
  |=======================================                                                                 |  38%
  |                                                                                                              
  |========================================                                                                |  38%
  |                                                                                                              
  |========================================                                                                |  39%
  |                                                                                                              
  |=========================================                                                               |  39%
  |                                                                                                              
  |=========================================                                                               |  40%
  |                                                                                                              
  |==========================================                                                              |  40%
  |                                                                                                              
  |==========================================                                                              |  41%
  |                                                                                                              
  |===========================================                                                             |  41%
  |                                                                                                              
  |===========================================                                                             |  42%
  |                                                                                                              
  |============================================                                                            |  42%
  |                                                                                                              
  |============================================                                                            |  43%
  |                                                                                                              
  |=============================================                                                           |  43%
  |                                                                                                              
  |=============================================                                                           |  44%
  |                                                                                                              
  |==============================================                                                          |  44%
  |                                                                                                              
  |==============================================                                                          |  45%
  |                                                                                                              
  |===============================================                                                         |  45%
  |                                                                                                              
  |===============================================                                                         |  46%
  |                                                                                                              
  |================================================                                                        |  46%
  |                                                                                                              
  |================================================                                                        |  47%
  |                                                                                                              
  |=================================================                                                       |  47%
  |                                                                                                              
  |==================================================                                                      |  48%
  |                                                                                                              
  |==================================================                                                      |  49%
  |                                                                                                              
  |===================================================                                                     |  49%
  |                                                                                                              
  |====================================================                                                    |  50%
  |                                                                                                              
  |=====================================================                                                   |  51%
  |                                                                                                              
  |======================================================                                                  |  51%
  |                                                                                                              
  |======================================================                                                  |  52%
  |                                                                                                              
  |=======================================================                                                 |  53%
  |                                                                                                              
  |========================================================                                                |  53%
  |                                                                                                              
  |========================================================                                                |  54%
  |                                                                                                              
  |=========================================================                                               |  54%
  |                                                                                                              
  |=========================================================                                               |  55%
  |                                                                                                              
  |==========================================================                                              |  55%
  |                                                                                                              
  |==========================================================                                              |  56%
  |                                                                                                              
  |===========================================================                                             |  56%
  |                                                                                                              
  |===========================================================                                             |  57%
  |                                                                                                              
  |============================================================                                            |  57%
  |                                                                                                              
  |============================================================                                            |  58%
  |                                                                                                              
  |=============================================================                                           |  58%
  |                                                                                                              
  |=============================================================                                           |  59%
  |                                                                                                              
  |==============================================================                                          |  59%
  |                                                                                                              
  |==============================================================                                          |  60%
  |                                                                                                              
  |===============================================================                                         |  60%
  |                                                                                                              
  |===============================================================                                         |  61%
  |                                                                                                              
  |================================================================                                        |  61%
  |                                                                                                              
  |================================================================                                        |  62%
  |                                                                                                              
  |=================================================================                                       |  62%
  |                                                                                                              
  |=================================================================                                       |  63%
  |                                                                                                              
  |==================================================================                                      |  63%
  |                                                                                                              
  |==================================================================                                      |  64%
  |                                                                                                              
  |===================================================================                                     |  64%
  |                                                                                                              
  |===================================================================                                     |  65%
  |                                                                                                              
  |====================================================================                                    |  65%
  |                                                                                                              
  |====================================================================                                    |  66%
  |                                                                                                              
  |=====================================================================                                   |  66%
  |                                                                                                              
  |=====================================================================                                   |  67%
  |                                                                                                              
  |======================================================================                                  |  67%
  |                                                                                                              
  |======================================================================                                  |  68%
  |                                                                                                              
  |=======================================================================                                 |  68%
  |                                                                                                              
  |=======================================================================                                 |  69%
  |                                                                                                              
  |========================================================================                                |  69%
  |                                                                                                              
  |========================================================================                                |  70%
  |                                                                                                              
  |=========================================================================                               |  70%
  |                                                                                                              
  |=========================================================================                               |  71%
  |                                                                                                              
  |==========================================================================                              |  71%
  |                                                                                                              
  |==========================================================================                              |  72%
  |                                                                                                              
  |===========================================================================                             |  72%
  |                                                                                                              
  |===========================================================================                             |  73%
  |                                                                                                              
  |============================================================================                            |  73%
  |                                                                                                              
  |=============================================================================                           |  74%
  |                                                                                                              
  |==============================================================================                          |  75%
  |                                                                                                              
  |===============================================================================                         |  76%
  |                                                                                                              
  |================================================================================                        |  77%
  |                                                                                                              
  |=================================================================================                       |  78%
  |                                                                                                              
  |==================================================================================                      |  78%
  |                                                                                                              
  |==================================================================================                      |  79%
  |                                                                                                              
  |===================================================================================                     |  79%
  |                                                                                                              
  |===================================================================================                     |  80%
  |                                                                                                              
  |====================================================================================                    |  80%
  |                                                                                                              
  |====================================================================================                    |  81%
  |                                                                                                              
  |=====================================================================================                   |  81%
  |                                                                                                              
  |=====================================================================================                   |  82%
  |                                                                                                              
  |======================================================================================                  |  82%
  |                                                                                                              
  |======================================================================================                  |  83%
  |                                                                                                              
  |=======================================================================================                 |  83%
  |                                                                                                              
  |=======================================================================================                 |  84%
  |                                                                                                              
  |========================================================================================                |  84%
  |                                                                                                              
  |========================================================================================                |  85%
  |                                                                                                              
  |=========================================================================================               |  85%
  |                                                                                                              
  |=========================================================================================               |  86%
  |                                                                                                              
  |==========================================================================================              |  86%
  |                                                                                                              
  |==========================================================================================              |  87%
  |                                                                                                              
  |===========================================================================================             |  87%
  |                                                                                                              
  |===========================================================================================             |  88%
  |                                                                                                              
  |============================================================================================            |  88%
  |                                                                                                              
  |============================================================================================            |  89%
  |                                                                                                              
  |=============================================================================================           |  89%
  |                                                                                                              
  |=============================================================================================           |  90%
  |                                                                                                              
  |==============================================================================================          |  90%
  |                                                                                                              
  |==============================================================================================          |  91%
  |                                                                                                              
  |===============================================================================================         |  91%
  |                                                                                                              
  |===============================================================================================         |  92%
  |                                                                                                              
  |================================================================================================        |  92%
  |                                                                                                              
  |================================================================================================        |  93%
  |                                                                                                              
  |=================================================================================================       |  93%
  |                                                                                                              
  |=================================================================================================       |  94%
  |                                                                                                              
  |==================================================================================================      |  94%
  |                                                                                                              
  |==================================================================================================      |  95%
  |                                                                                                              
  |===================================================================================================     |  95%
  |                                                                                                              
  |===================================================================================================     |  96%
  |                                                                                                              
  |====================================================================================================    |  96%
  |                                                                                                              
  |====================================================================================================    |  97%
  |                                                                                                              
  |=====================================================================================================   |  97%
  |                                                                                                              
  |=====================================================================================================   |  98%
  |                                                                                                              
  |======================================================================================================  |  98%
  |                                                                                                              
  |======================================================================================================= |  99%
  |                                                                                                              
  |========================================================================================================| 100%

Now let’s make a plot to look at the distribution of scores from random gene sets. First we need to get this data in an appropriate format for ggplot2.

# The random results are a matrix
random_long_df <- random_gsva_results %>%
  data.frame() %>%
  # Gene set names are rownames
  tibble::rownames_to_column("gene_set") %>%
  # Get into long format
  tidyr::pivot_longer(cols = -gene_set, 
                      names_to = "Kids_First_Biospecimen_ID",
                      values_to = "gsva_score") %>%
  # Remove the .version added by make.names()
  dplyr::mutate(gene_set = stringr::str_remove(gene_set, "\\..*")) %>%
  # Add a column that keeps track of the gene set size
  dplyr::mutate(gene_set_size = stringr::word(gene_set, 2, sep = "_")) %>%
  # We want to plot smallest no. genes -> largest no. genes
  dplyr::mutate(gene_set_size = factor(gene_set_size, 
                                       levels = c(15, 25, 50, 100, 250, 500)))

Let’s make a violin plot so we can look at the distribution of scores by gene set size.

# Violin plot comparing GSVA scores of different random gene set sizes
random_long_df %>%
  ggplot2::ggplot(ggplot2::aes(x = gene_set_size, 
                               y = gsva_score)) +
  # Make a violin plot that's a pretty blue!
  ggplot2::geom_violin(fill = "#99CCFF", alpha = 0.5) +
  # Add a point with the mean value
  ggplot2::stat_summary(
    geom = "point",
    fun = "mean",
    # Change the aesthetics of the points
    size = 3,
    color = "#0066CC",
    shape = 18
  ) +
  # Flip the axes
  ggplot2::coord_flip() +
  ggplot2::labs(title = "Random Gene Set GSVA Scores",
                x = "gene set size",
                y = "GSVA score") +
  ggplot2::theme_bw()

What do you notice about these distributions? How might you use this information to inform your interpretation of GSVA scores?

How can you use these scores?

If you did have groups information for your samples, you could test for pathway scores differences between groups. Here’s an example of that from the OpenPBTA project itself!

You can also visualize this matrix in a heatmap. Here’s a figure from the OpenPBTA project, where the middle panel is a heatmap of GSVA scores that were significantly different between histologies.

Code here.

Write results to file

gsva_results %>%
  as.data.frame() %>%
  tibble::rownames_to_column("pathway") %>%
  readr::write_tsv(file = gsva_results_file)

Session Info

sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8    
 [5] LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C             
 [9] LC_ADDRESS=C           LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] GSVA_1.38.2    magrittr_2.0.1

loaded via a namespace (and not attached):
 [1] MatrixGenerics_1.2.0        Biobase_2.50.0              httr_1.4.2                 
 [4] tidyr_1.1.2                 bit64_4.0.5                 jsonlite_1.7.2             
 [7] assertthat_0.2.1            BiocManager_1.30.10         stats4_4.0.3               
[10] blob_1.2.1                  renv_0.12.5-2               GenomeInfoDbData_1.2.4     
[13] yaml_2.2.1                  pillar_1.4.7                RSQLite_2.2.3              
[16] lattice_0.20-41             glue_1.4.2                  limma_3.46.0               
[19] digest_0.6.27               GenomicRanges_1.42.0        XVector_0.30.0             
[22] colorspace_2.0-0            htmltools_0.5.1.1           Matrix_1.3-2               
[25] GSEABase_1.52.1             XML_3.99-0.5                pkgconfig_2.0.3            
[28] zlibbioc_1.36.0             purrr_0.3.4                 xtable_1.8-4               
[31] mvtnorm_1.1-1               scales_1.1.1                BiocParallel_1.24.1        
[34] emmeans_1.5.3               tibble_3.0.5                annotate_1.68.0            
[37] generics_0.1.0              farver_2.0.3                IRanges_2.24.1             
[40] ggplot2_3.3.3               ellipsis_0.3.1              SummarizedExperiment_1.20.0
[43] BiocGenerics_0.36.0         cli_2.2.0                   crayon_1.3.4               
[46] memoise_1.1.0               estimability_1.3            evaluate_0.14              
[49] fansi_0.4.2                 nlme_3.1-151                graph_1.68.0               
[52] tools_4.0.3                 hms_1.0.0                   lifecycle_0.2.0            
[55] matrixStats_0.57.0          stringr_1.4.0               S4Vectors_0.28.1           
[58] fftw_1.0-6                  munsell_0.5.0               DelayedArray_0.16.2        
[61] AnnotationDbi_1.52.0        compiler_4.0.3              GenomeInfoDb_1.26.2        
[64] rlang_0.4.10                grid_4.0.3                  RCurl_1.98-1.2             
[67] rstudioapi_0.13             bitops_1.0-6                labeling_0.4.2             
[70] rmarkdown_2.6               qusage_2.24.0               gtable_0.3.0               
[73] DBI_1.1.1                   R6_2.5.0                    knitr_1.30                 
[76] dplyr_1.0.3                 bit_4.0.4                   readr_1.4.0                
[79] stringi_1.5.3               parallel_4.0.3              Rcpp_1.0.6                 
[82] vctrs_0.3.6                 tidyselect_1.1.0            xfun_0.20                  
[85] coda_0.19-4                
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzIChHU1ZBKSIKb3V0cHV0OiAKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjAKLS0tCgpTbyBmYXIgZXZlcnkgcGF0aHdheSBhbmFseXNpcyBtZXRob2Qgd2UndmUgY292ZXJlZCByZWxpZXMgb24gc29tZSBpbmZvcm1hdGlvbiBhYm91dCBncm91cHMgb2Ygc2FtcGxlcyBpbiBvdXIgZGF0YS4KRm9yIG92ZXItcmVwcmVzZW50YXRpb24gYW5hbHlzaXMgKE9SQSksIHdlIGNyZWF0ZWQgZ2VuZSBzZXRzIGZyb20gdHdvIGRpZmZlcmVudCB0d28gZ3JvdXAgY29tcGFyaXNvbnMuCkluIHRoZSBHZW5lIFNldCBFbnJpY2htZW50IEFuYWx5c2lzIChHU0VBKSBleGFtcGxlLCB3ZSB1c2VkIHN0YXRpc3RpY3MgZnJvbSBhIGRpZmZlcmVudGlhbCBnZW5lIGV4cHJlc3Npb24gKERHRSkgYW5hbHlzaXMgd2hlcmUgd2UgY29tcGFyZWQgX01ZQ05fIGFtcGxpZmllZCBjZWxsIGxpbmVzIHRvIG5vbi1hbXBsaWZpZWQgY2VsbCBsaW5lczsgd2UgbmVlZGVkIHRoYXQgYW1wbGlmaWNhdGlvbiBzdGF0dXMgaW5mb3JtYXRpb24uCgpXaGF0IGlmIHdlJ3JlIGxlc3Mgc3VyZSBhYm91dCBncm91cHMgaW4gb3VyIGRhdGEgb3Igd2Ugd2FudCB0byBhbmFseXplIG91ciBkYXRhIGluIGEgbW9yZSB1bnN1cGVydmlzZWQgbWFubmVyPwoKSW4gdGhpcyBub3RlYm9vayB3ZSB3aWxsIGNvdmVyIGEgbWV0aG9kIGNhbGxlZCBHZW5lIFNldCBWYXJpYXRpb24gQW5hbHlzaXMgKEdTVkEpIChbSMOkbnplbG1hbm4sIENhc3RlbG8sIGFuZCBHdWlubmV5LiAyMDEzXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykpIHRoYXQgYWxsb3dzIHVzIHRvIGNhbGN1bGF0ZSBnZW5lIHNldCBvciBwYXRod2F5IHNjb3JlcyBvbiBhIHBlci1zYW1wbGUgYmFzaXMuCgpXZSBsaWtlIHRoaXMgcXVvdGUgZnJvbSB0aGUgR1NWQSBwYXBlciAoW0jDpG56ZWxtYW5uIF9ldCBhbC5fIDIwMTNdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMTg2LzE0NzEtMjEwNS0xNC03KSkgdG8gc2V0IHRoZSBzdGFnZToKCj4gV2hpbGUgW2dlbmUgc2V0IGVucmljaG1lbnRdIG1ldGhvZHMgYXJlIGdlbmVyYWxseSByZWdhcmRlZCBhcyBlbmQgcG9pbnRzIG9mIGEgYmlvaW5mb3JtYXRpYyBhbmFseXNpcywgR1NWQSBjb25zdGl0dXRlcyBhIHN0YXJ0aW5nIHBvaW50IHRvIGJ1aWxkIHBhdGh3YXktY2VudHJpYyBtb2RlbHMgb2YgYmlvbG9neS4KClJhdGhlciB0aGFuIGNvbnRleHR1YWxpemluZyBzb21lIHJlc3VsdHMgeW91IF9hbHJlYWR5IGhhdmVfIGZyb20gYW5vdGhlciBhbmFseXNpcyBsaWtlIERHRSwgR1NWQSBpcyBkZXNpZ25lZCB0byBwcm92aWRlIGFuIGVzdGltYXRlIG9mIHBhdGh3YXkgdmFyaWF0aW9uIGZvciBlYWNoIG9mIHRoZSBzYW1wbGVzIGluIGFuIGV4cGVyaW1lbnQuCk5vdGUgdGhhdCB0aGVzZSBzY29yZXMgd2lsbCBkZXBlbmQgb24gdGhlIHNhbXBsZXMgaW5jbHVkZWQgaW4gdGhlIGRhdGFzZXQgd2hlbiB5b3UgcnVuIEdTVkE7IGlmIHlvdSBhZGRlZCBtb3JlIHNhbXBsZXMgYW5kIHJlcmFuIEdTVkEsIHlvdSB3b3VsZCBleHBlY3QgdGhlIHNjb3JlcyB0byBjaGFuZ2UuCgojIyMjIE90aGVyIHJlc291cmNlcwoKKiBbTWFsaG90cmEuIF9EZWNvZGluZyBHZW5lIFNldCBWYXJpYXRpb24gQW5hbHlzaXNfLiAoMjAxOCldKGh0dHBzOi8vdG93YXJkc2RhdGFzY2llbmNlLmNvbS9kZWNvZGluZy1nZW5lLXNldC12YXJpYXRpb24tYW5hbHlzaXMtODE5M2EwY2ZkYTMpCgojIyBTZXQgdXAKCiMjIyBMaWJyYXJpZXMKCmBgYHtyIGxpYnJhcmllc30KIyBQaXBlcwpsaWJyYXJ5KG1hZ3JpdHRyKQojIEdlbmUgU2V0IFZhcmlhdGlvbiBBbmFseXNpcwpsaWJyYXJ5KEdTVkEpCmBgYAoKIyMjIERpcmVjdG9yaWVzIGFuZCBmaWxlcwoKIyMjIyBEaXJlY3RvcmllcwoKYGBge3IgZGlyZWN0b3JpZXN9CiMgV2UgaGF2ZSBzb21lIG1lZHVsbG9ibGFzdG9tYSBkYXRhIGZyb20gdGhlIE9wZW5QQlRBIHByb2plY3QgdGhhdCB3ZSd2ZSAKIyBwcmVwYXJlZCBhaGVhZCBvZiB0aW1lCmlucHV0X2RpciA8LSBmaWxlLnBhdGgoImRhdGEiLCAib3Blbi1wYnRhIikKCiMgQ3JlYXRlIGEgZGlyZWN0b3J5IHNwZWNpZmljYWxseSBmb3IgdGhlIHJlc3VsdHMgdXNpbmcgdGhpcyBkYXRhc2V0Cm91dHB1dF9kaXIgPC0gZmlsZS5wYXRoKCJyZXN1bHRzIiwgIm9wZW4tcGJ0YSIpIAppZiAoIWRpci5leGlzdHMob3V0cHV0X2RpcikpIHsKICBkaXIuY3JlYXRlKG91dHB1dF9kaXIsIHJlY3Vyc2l2ZSA9IFRSVUUpCn0KYGBgCgojIyMjIElucHV0CgpXZSBoYXZlIFZTVCB0cmFuc2Zvcm1lZCBSTkEtc2VxIGRhdGEsIGFubm90YXRlZCB3aXRoIGdlbmUgc3ltYm9scywgdGhhdCBoYXMgYmVlbiBjb2xsYXBzZWQgc3VjaCB0aGF0IHRoZXJlIGFyZSBubyBkdXBsaWNhdGVkIGdlbmUgaWRlbnRpZmllcnMgKHNlZSBgc2V0dXBgKS4KCmBgYHtyIGlucHV0X2ZpbGV9CnJuYXNlcV9maWxlIDwtIGZpbGUucGF0aChpbnB1dF9kaXIsICJtZWR1bGxvYmxhc3RvbWFfdnN0X2NvbGxhcHNlZC50c3YiKQpgYGAKCiMjIyMgT3V0cHV0CgpgYGB7ciBvdXRwdXRfZmlsZSwgbGl2ZSA9IFRSVUV9CmdzdmFfcmVzdWx0c19maWxlIDwtIGZpbGUucGF0aChvdXRwdXRfZGlyLCAibWVkdWxsb2JsYXN0b21hX2dzdmFfcmVzdWx0cy50c3YiKQpgYGAKCiMjIEdlbmUgc2V0cwoKVGhlIGZ1bmN0aW9uIHRoYXQgd2Ugd2lsbCB1c2UgdG8gcnVuIEdTVkEgd2FudHMgdGhlIGdlbmUgc2V0cyB0byBiZSBpbiBhIGxpc3QsIHJhdGhlciB0aGFuIGEgdGlkeSBkYXRhIGZyYW1lIHRoYXQgd2UgdXNlZCB3aXRoIGBjbHVzdGVyUHJvZmlsZXJgIChhbHRob3VnaCBpdCBkb2VzIGFjY2VwdCBvdGhlciBmb3JtYXRzKS4KCldlJ3JlIGdvaW5nIHRvIHRha2UgdGhpcyBvcHBvcnR1bml0eSB0byBpbnRyb2R1Y2UgYSBkaWZmZXJlbnQgZm9ybWF0IHRoYXQgZ2VuZSBzZXRzIGFyZSBvZnRlbiBkaXN0cmlidXRlZCBpbiBjYWxsZWQgW0dNVCAoR2VuZSBNYXRyaXggVHJhbnNwb3NlZCldKGh0dHBzOi8vc29mdHdhcmUuYnJvYWRpbnN0aXR1dGUub3JnL2NhbmNlci9zb2Z0d2FyZS9nc2VhL3dpa2kvaW5kZXgucGhwL0RhdGFfZm9ybWF0cyNHTVQ6X0dlbmVfTWF0cml4X1RyYW5zcG9zZWRfZmlsZV9mb3JtYXRfLjI4LjJBLmdtdC4yOSkuCgpXZSdyZSBnb2luZyB0byByZWFkIGluIHRoZSBIYWxsbWFyayBjb2xsZWN0aW9uIGZpbGUgZGlyZWN0bHkgZnJvbSBbTVNpZ0RCXShodHRwczovL3d3dy5nc2VhLW1zaWdkYi5vcmcvZ3NlYS9tc2lnZGIvaW5kZXguanNwKSwgcmF0aGVyIHRoYW4gdXNpbmcgYG1zaWdkYnJgIGxpa2Ugd2UgZGlkIGluIGVhcmxpZXIgbm90ZWJvb2tzLgoKVGhlIFJOQS1zZXEgZGF0YSB1c2VzIGdlbmUgc3ltYm9scywgc28gd2UgbmVlZCBnZW5lIHNldHMgdGhhdCB1c2UgZ2VuZSBzeW1ib2xzLCB0b28uCgpgYGB7ciBnbXR9CiMgUiBjYW4gb2Z0ZW4gcmVhZCBpbiBkYXRhIGZyb20gYSBVUkwKaGFsbG1hcmtzX3VybCA8LSAiaHR0cHM6Ly9kYXRhLmJyb2FkaW5zdGl0dXRlLm9yZy9nc2VhLW1zaWdkYi9tc2lnZGIvcmVsZWFzZS83LjEvaC5hbGwudjcuMS5zeW1ib2xzLmdtdCIKCiMgUXVTQUdFIGlzIGFub3RoZXIgcGF0aHdheSBhbmFseXNpcyBtZXRob2QsIHRoZSBxdXNhZ2UgcGFja2FnZSBoYXMgYSBmdW5jdGlvbgojIGZvciByZWFkaW5nIEdNVCBmaWxlcyBhbmQgdHVybmluZyB0aGVtIGludG8gYSBsaXN0CmhhbGxtYXJrc19saXN0IDwtIHF1c2FnZTo6cmVhZC5nbXQoaGFsbG1hcmtzX3VybCkKYGBgCgpXaGF0IGRvZXMgdGhpcyBsaXN0IGxvb2sgbGlrZT8KCmBgYHtyIGhlYWRfaGFsbG1hcmssIGV2YWwgPSBGQUxTRX0KaGVhZChoYWxsbWFya3NfbGlzdCkKYGBgCgojIyBSTkEtc2VxIGRhdGEKCldlIGhhdmUgVlNUIHRyYW5zZm9ybWVkIFJOQS1zZXEgZGF0YSwgd2hpY2ggaXMgb24gYSBsb2cyLWxpa2Ugc2NhbGUuClRoZXNlIGRhdGEgYXJlIGZyb20gdGhlIE9wZW4gUGVkaWF0cmljIEJyYWluIFR1bW9yIEF0bGFzIChPcGVuUEJUQSkKT3BlblBCVEEgaXMgYSBjb2xsYWJvcmF0aXZlIHByb2plY3Qgb3JnYW5pemVkIGJ5IHRoZSBDQ0RMIGFuZCB0aGUgQ2VudGVyIGZvciBEYXRhLURyaXZlbiBEaXNjb3ZlcnkgaW4gQmlvbWVkaWNpbmUgKEQzYikgYXQgdGhlIENoaWxkcmVuJ3MgSG9zcGl0YWwgb2YgUGhpbGFkZWxwaGlhIGNvbmR1Y3RlZCBvcGVubHkgb24gR2l0SHViLgoKWW91IGNhbiByZWFkIG1vcmUgYWJvdXQgdGhlIHByb2plY3QgW2hlcmVdKGh0dHBzOi8vZ2l0aHViLmNvbS9hbGV4c2xlbW9uYWRlL29wZW5wYnRhLWFuYWx5c2lzLyNvcGVucGJ0YS1hbmFseXNpcykuCgpXZSdyZSBvbmx5IHdvcmtpbmcgd2l0aCB0aGUgbWVkdWxsb2JsYXN0b21hIHNhbXBsZXMgaW4gdGhpcyBleGFtcGxlLgoKYGBge3IgcmVhZF9pbl9ybmFzZXF9CnJuYXNlcV9kZiA8LSByZWFkcjo6cmVhZF90c3Yocm5hc2VxX2ZpbGUpCmBgYAoKYGBge3IgcGVla19ybmFzZXEsIGxpdmUgPSBUUlVFfQojIFdoYXQgZG9lcyB0aGUgUk5BLXNlcSBkYXRhIGZyYW1lIGxvb2sgbGlrZT8Kcm5hc2VxX2RmWzE6NSwgMTo1XQpgYGAKCkZvciBHU1ZBLCB3ZSBuZWVkIGEgbWF0cml4LgoKYGBge3Igcm5hc2VxX21hdCwgbGl2ZSA9IFRSVUV9CnJuYXNlcV9tYXQgPC0gcm5hc2VxX2RmICU+JQogIHRpYmJsZTo6Y29sdW1uX3RvX3Jvd25hbWVzKCJnZW5lX3N5bWJvbCIpICU+JQogIGFzLm1hdHJpeCgpCmBgYAoKKk5vdGU6IElmIHdlIGhhZCBkdXBsaWNhdGUgZ2VuZSBzeW1ib2xzIGhlcmUsIHdlIGNvdWxkbid0IHNldCB0aGVtIGFzIHJvd25hbWVzLioKCiMjIEdTVkEKCiFbXShkaWFncmFtcy9oYW56ZWxtYW5uX2ZpZzEuanBnKQoKKipGaWd1cmUgMSBmcm9tIFtIw6RuemVsbWFubiwgQ2FzdGVsbywgYW5kIEd1aW5uZXkuICgyMDEzKV0oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpLioqCgpZb3UgbWF5IG5vdGljZSB0aGF0IEdTVkEgaGFzIHNvbWUgY29tbW9uYWxpdGllcyB3aXRoIEdTRUEuClJhdGhlciB0aGFuIHJhbmtpbmcgZ2VuZXMgYmFzZWQgb24gc29tZSBzdGF0aXN0aWMgX3dlXyBzZWxlY3RlZCBhaGVhZCBvZiB0aW1lLCBHU1ZBIGZpdHMgYSBtb2RlbCBhbmQgcmFua3MgZ2VuZXMgYmFzZWQgb24gdGhlaXIgZXhwcmVzc2lvbiBsZXZlbCByZWxhdGl2ZSB0byB0aGUgc2FtcGxlIGRpc3RyaWJ1dGlvbi4KVGhpcyBpcyBhIHdheSBvZiBhc2tpbmcgaWYgYSBnZW5lIF9pXyBpcyBoaWdobHkgb3IgbG93bHkgZXhwcmVzc2VkIGluIGEgc2FtcGxlIF9qXyBpbiB0aGUgY29udGV4dCBvZiB0aGlzIGV4cGVyaW1lbnQgYW5kIHJhbmtpbmcgYWNjb3JkaW5nbHkgKFtIw6RuemVsbWFubiBfZXQgYWwuXyAyMDEzXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykpLgpUaGUgcGF0aHdheS1sZXZlbCBzY29yZSBjYWxjdWxhdGVkIGlzIGEgd2F5IG9mIGFza2luZyBob3cgZ2VuZXMgX3dpdGhpbl8gYSBnZW5lIHNldCB2YXJ5IGFzIGNvbXBhcmVkIHRvIGdlbmVzIHRoYXQgYXJlIF9vdXRzaWRlXyBvZiB0aGF0IGdlbmUgc2V0IChbTWFsaG90cmEuIDIwMThdKGh0dHBzOi8vdG93YXJkc2RhdGFzY2llbmNlLmNvbS9kZWNvZGluZy1nZW5lLXNldC12YXJpYXRpb24tYW5hbHlzaXMtODE5M2EwY2ZkYTMpKS4gCihUaGlzIGlzIHNvbWV0aW1lcyBjYWxsZWQgYSBjb21wZXRpdGl2ZSB0ZXN0IGluIGdlbmUgc2V0IGVucmljaG1lbnQgbGl0ZXJhdHVyZS4pClRoZSBpbnR1aXRpb24gaGVyZSBpcyB0aGF0IHdlIHdpbGwgZ2V0IHBhdGh3YXktbGV2ZWwgc2NvcmVzIGZvciBlYWNoIHNhbXBsZSB0aGF0IGluZGljYXRlIGlmIGdlbmVzIGluIGEgcGF0aHdheSB2YXJ5IGNvbmNvcmRhbnRseSBpbiBvbmUgZGlyZWN0aW9uIChvdmVyZXhwcmVzc2VkIG9yIHVuZGVyZXhwcmVzc2VkIHJlbGF0aXZlIHRvIHRoZSBvdmVyYWxsIHBvcHVsYXRpb24pIChbSMOkbnplbG1hbm4gX2V0IGFsLl8gMjAxM10oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpKS4KClRoZSBvdXRwdXQgaXMgYSBnZW5lIHNldCBieSBzYW1wbGUgbWF0cml4IG9mIEdTVkEgc2NvcmVzLgoKIyMjIFBlcmZvcm0gR1NWQQoKYGBge3IgcnVuX2dzdmF9CmdzdmFfcmVzdWx0cyA8LSBnc3ZhKHJuYXNlcV9tYXQsIAogICAgICAgICAgICAgICAgICAgICBoYWxsbWFya3NfbGlzdCwKICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gImdzdmEiLAogICAgICAgICAgICAgICAgICAgICAjIEFwcHJvcHJpYXRlIGZvciBvdXIgdHJhbnNmb3JtZWQgZGF0YSBvbiBsb2cyLWxpa2Ugc2NhbGUKICAgICAgICAgICAgICAgICAgICAga2NkZiA9ICJHYXVzc2lhbiIsICAKICAgICAgICAgICAgICAgICAgICAgIyBNaW5pbXVtIGdlbmUgc2V0IHNpemUKICAgICAgICAgICAgICAgICAgICAgbWluLnN6ID0gMTUsIAogICAgICAgICAgICAgICAgICAgICAjIE1heGltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICBtYXguc3ogPSA1MDAsCiAgICAgICAgICAgICAgICAgICAgICMgQ29tcHV0ZSBHYXVzc2lhbi1kaXN0cmlidXRlZCBzY29yZXMKICAgICAgICAgICAgICAgICAgICAgbXguZGlmZiA9IFRSVUUpCmBgYAoKKipOb3RlOiB0aGUgYGdzdmEoKWAgZG9jdW1lbnRhdGlvbiBzYXlzIHdlIGNhbiB1c2UgYGtjZGYgPSAiR2F1c3NpYW4iYCBpZiB3ZSBoYWQgUk5BLXNlcSBsb2ctQ1BNcywgbG9nLVJQS01zIG9yIGxvZy1UUE1zLCBidXQgd2Ugd291bGQgdXNlIGBrY2RmID0gIlBvaXNzb24iYCBvbiBpbnRlZ2VyIGNvdW50cy4qKgoKYGBge3IgZ3N2YV9wZWVrfQojIExldCdzIGV4cGxvcmUgd2hhdCB0aGUgb3V0cHV0IG9mIGdzdmEoKSBsb29rcyBsaWtlCmdzdmFfcmVzdWx0c1sxOjUsIDE6NV0KYGBgCgojIyMgQSBub3RlIG9uIGdlbmUgc2V0IHNpemUKIApPZnRlbiB0aGUgc2NvcmVzIG9mIGdlbmUgc2V0IGVucmljaG1lbnQgbWV0aG9kcyBhcmUgbm90IGNvbXBhcmFibGUgYmV0d2VlbiBnZW5lIHNldHMgb2YgZGlmZmVyZW50IHNpemVzLgpMZXQncyBkbyBhbiBleHBlcmltZW50IHVzaW5nIHJhbmRvbWx5IGdlbmVyYXRlZCBnZW5lIHNldHMgdG8gZXhwbG9yZSB0aGlzIGlkZWEgYSBiaXQgbW9yZS4KCldlIG5lZWQgdG8gZ2V0IGEgY29sbGVjdGlvbiBvZiBhbGwgcG9zc2libGUgZ2VuZXMgd2Ugd2lsbCBzYW1wbGUgZnJvbSB0byBjcmVhdGUgcmFuZG9tIGdlbmUgc2V0cy4KQmVjYXVzZSB3ZSdyZSBkb2luZyBzb21lIHJhbmRvbSBzYW1wbGluZywgd2UgbmVlZCB0byBzZXQgYSBzZWVkIGZvciB0aGlzIHRvIGJlIHJlcHJvZHVjaWJsZS4KCmBgYHtyIHJhbmRvbV9zZXR1cH0KIyBVc2UgYWxsIHRoZSBnZW5lIHN5bWJvbHMgaW4gdGhlIGRhdGFzZXQgYXMgdGhlIHBvb2wgb2YgcG9zc2libGUgZ2VuZXMKYWxsX2dlbmVzIDwtIHJvd25hbWVzKHJuYXNlcV9tYXQpCgojIFNldCBhIHNlZWQgZm9yIHJlcHJvZHVjaWJpbGl0eQpzZXQuc2VlZCgyMDIwKQpgYGAKCk91ciBtaW5pbXVtIGdlbmUgc2V0IHNpemUgZWFybGllciB3YXMgMTUgZ2VuZXMgYW5kIG91ciBtYXhpbXVtIGdlbmUgc2V0IHNpemUgd2FzIDUwMCBnZW5lcy4gV2UnbGwgdXNlIHRoZSBzYW1lIG1pbmltdW0gYW5kIG1heGltdW0gdmFsdWVzIGZvciBvdXIgcmFuZG9tIGdlbmUgc2V0cyBhbmQgc29tZSB2YWx1ZXMgaW4gYmV0d2Vlbi4KCmBgYHtyIGdlbmVfc2V0X3NpemVzfQojIE1ha2UgYSBsaXN0IG9mIGludGVnZXJzIHRoYXQgaW5kaWNhdGUgdGhlIHJhbmRvbSBnZW5lIHNldCBzaXplcwpnZW5lX3NldF9zaXplIDwtIGxpc3QoMTUsIDI1LCA1MCwgMTAwLCAyNTAsIDUwMCkKYGBgCgpGb3IgZWFjaCBnZW5lIHNldCBzaXplLCB3ZSB3aWxsIGdlbmVyYXRlIDEwMCByYW5kb20gZ2VuZSBzZXRzCgpgYGB7ciByYW5kb21fZ2VuZV9zZXRzfQojIFNldCBudW1iZXIgb2YgcmVwbGljYXRlcwpucmVwcyA8LSAgMTAwCiMgR2VuZXJhdGUgMTAwIHJhbmRvbSBnZW5lIHNldHMgb2YgZWFjaCBzaXplCnJhbmRvbV9nZW5lX3NldHMgPC0gCiAgcHVycnI6Om1hcCgKICAgIHJlcChnZW5lX3NldF9zaXplLCBucmVwcyksICAjIFJlcGVhdCBnZW5lIHNpemVzIHNvIHdlIHJ1biBgbnJlcHNgIHRpbWVzIAogICAgIyBTYW1wbGUgdGhlIHZlY3RvciBvZiBhbGwgZ2VuZXMsIGNob29zaW5nIHRoZSBudW1iZXIgb2YgaXRlbXMgc3BlY2lmaWVkCiAgICAjIGluIHRoZSBlbGVtZW50IG9mIGdlbmUgc2V0IHNpemUKICAgIH4gYmFzZTo6c2FtcGxlKHggPSBhbGxfZ2VuZXMsCiAgICAgICAgICAgICAgICAgICBzaXplID0gLngpCiAgKQpgYGAKClRoZSBIYWxsbWFya3MgbGlzdCB3ZSB1c2VkIGVhcmxpZXIgc3RvcmVkIHRoZSBnZW5lIHNldCBuYW1lcyBhcyB0aGUgbmFtZSBvZiB0aGUgbGlzdCwgc28gbGV0J3MgYWRkIG5hbWVzIHRvIG91ciByYW5kb20gZ2VuZSBzZXRzIHRoYXQgaW5kaWNhdGUgd2hhdCBzaXplIHRoZXkgYXJlIGFuZCBzbyBgZ3N2YSgpYCBkb2Vzbid0IGdldCB1cHNldC4KCmBgYHtyIG5hbWVfcmFuZG9tX2dlbmVfc2V0c30KIyBXZSB3aWxsIGluY2x1ZGUgdGhlIHNpemUgb2YgdGhlIGdlbmUgc2V0IGluIHRoZSBnZW5lIHNldCBuYW1lIAojIFN0YXJ0IGJ5IHRha2luZyB0aGUgbGVuZ3RoIG9mIGVhY2ggcGF0aHdheSBhbmQgYXBwZW5kaW5nICJwYXRod2F5XyIgdG8gdGhhdAojIG51bWJlcgpsZW5ndGhzX3ZlY3RvciA8LSBwdXJycjo6bWFwKHJhbmRvbV9nZW5lX3NldHMsIH4gbGVuZ3RoKC54KSkgJT4lCiAgcHVycnI6Om1hcCh+IHBhc3RlMCgicGF0aHdheV8iLCAueCkpICU+JQogICMgUmV0dXJuIGEgdmVjdG9yCiAgcHVycnI6OmZsYXR0ZW5fY2hyKCkKCiMgQWRkIHRoZSBuYW1lcyBpbiBsZW5ndGhzX3ZlY3RvciB0byB0aGUgbGlzdApyYW5kb21fZ2VuZV9zZXRzIDwtIHJhbmRvbV9nZW5lX3NldHMgJT4lIAogICMgbWFrZS5uYW1lcygpIGFwcGVuZHMgYSAidmVyc2lvbiIgaWYgc29tZXRoaW5nIGlzIG5vdCB1bmlxdWUKICBwdXJycjo6c2V0X25hbWVzKG5tID0gbWFrZS5uYW1lcyhsZW5ndGhzX3ZlY3RvciwgdW5pcXVlID0gVFJVRSkpCmBgYAogClJ1biBHU1ZBIG9uIG91ciBkYXRhc2V0IHdpdGggdGhlIHNhbWUgcGFyYW1ldGVycyBhcyBiZWZvcmUsIGJ1dCBub3cgd2l0aCByYW5kb20gZ2VuZSBzZXRzLgoKYGBge3IgcmFuZG9tX2dzdmEsIGxpdmUgPSBUUlVFfQpyYW5kb21fZ3N2YV9yZXN1bHRzIDwtIGdzdmEocm5hc2VxX21hdCwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICByYW5kb21fZ2VuZV9zZXRzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gImdzdmEiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBBcHByb3ByaWF0ZSBmb3Igb3VyIHRyYW5zZm9ybWVkIGRhdGEgb24gCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIGxvZzItbGlrZSBzY2FsZQogICAgICAgICAgICAgICAgICAgICAgICAgICAga2NkZiA9ICJHYXVzc2lhbiIsICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgTWluaW11bSBnZW5lIHNldCBzaXplCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBtaW4uc3ogPSAxNSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIE1heGltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4LnN6ID0gNTAwLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBDb21wdXRlIEdhdXNzaWFuLWRpc3RyaWJ1dGVkIHNjb3JlcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgbXguZGlmZiA9IFRSVUUpICAgICAgICAgICAgICAgICAgICAgICAgICAgIApgYGAKCk5vdyBsZXQncyBtYWtlIGEgcGxvdCB0byBsb29rIGF0IHRoZSBkaXN0cmlidXRpb24gb2Ygc2NvcmVzIGZyb20gcmFuZG9tIGdlbmUgc2V0cy4KRmlyc3Qgd2UgbmVlZCB0byBnZXQgdGhpcyBkYXRhIGluIGFuIGFwcHJvcHJpYXRlIGZvcm1hdCBmb3IgYGdncGxvdDJgLgoKYGBge3IgbG9uZ2VyX3JhbmRvbV9nc3ZhfQojIFRoZSByYW5kb20gcmVzdWx0cyBhcmUgYSBtYXRyaXgKcmFuZG9tX2xvbmdfZGYgPC0gcmFuZG9tX2dzdmFfcmVzdWx0cyAlPiUKICBkYXRhLmZyYW1lKCkgJT4lCiAgIyBHZW5lIHNldCBuYW1lcyBhcmUgcm93bmFtZXMKICB0aWJibGU6OnJvd25hbWVzX3RvX2NvbHVtbigiZ2VuZV9zZXQiKSAlPiUKICAjIEdldCBpbnRvIGxvbmcgZm9ybWF0CiAgdGlkeXI6OnBpdm90X2xvbmdlcihjb2xzID0gLWdlbmVfc2V0LCAKICAgICAgICAgICAgICAgICAgICAgIG5hbWVzX3RvID0gIktpZHNfRmlyc3RfQmlvc3BlY2ltZW5fSUQiLAogICAgICAgICAgICAgICAgICAgICAgdmFsdWVzX3RvID0gImdzdmFfc2NvcmUiKSAlPiUKICAjIFJlbW92ZSB0aGUgLnZlcnNpb24gYWRkZWQgYnkgbWFrZS5uYW1lcygpCiAgZHBseXI6Om11dGF0ZShnZW5lX3NldCA9IHN0cmluZ3I6OnN0cl9yZW1vdmUoZ2VuZV9zZXQsICJcXC4uKiIpKSAlPiUKICAjIEFkZCBhIGNvbHVtbiB0aGF0IGtlZXBzIHRyYWNrIG9mIHRoZSBnZW5lIHNldCBzaXplCiAgZHBseXI6Om11dGF0ZShnZW5lX3NldF9zaXplID0gc3RyaW5ncjo6d29yZChnZW5lX3NldCwgMiwgc2VwID0gIl8iKSkgJT4lCiAgIyBXZSB3YW50IHRvIHBsb3Qgc21hbGxlc3Qgbm8uIGdlbmVzIC0+IGxhcmdlc3Qgbm8uIGdlbmVzCiAgZHBseXI6Om11dGF0ZShnZW5lX3NldF9zaXplID0gZmFjdG9yKGdlbmVfc2V0X3NpemUsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsZXZlbHMgPSBjKDE1LCAyNSwgNTAsIDEwMCwgMjUwLCA1MDApKSkKCmBgYAoKTGV0J3MgbWFrZSBhIHZpb2xpbiBwbG90IHNvIHdlIGNhbiBsb29rIGF0IHRoZSBkaXN0cmlidXRpb24gb2Ygc2NvcmVzIGJ5IGdlbmUgc2V0IHNpemUuCgpgYGB7ciByYW5kb21fdmlvbGlufQojIFZpb2xpbiBwbG90IGNvbXBhcmluZyBHU1ZBIHNjb3JlcyBvZiBkaWZmZXJlbnQgcmFuZG9tIGdlbmUgc2V0IHNpemVzCnJhbmRvbV9sb25nX2RmICU+JQogIGdncGxvdDI6OmdncGxvdChnZ3Bsb3QyOjphZXMoeCA9IGdlbmVfc2V0X3NpemUsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeSA9IGdzdmFfc2NvcmUpKSArCiAgIyBNYWtlIGEgdmlvbGluIHBsb3QgdGhhdCdzIGEgcHJldHR5IGJsdWUhCiAgZ2dwbG90Mjo6Z2VvbV92aW9saW4oZmlsbCA9ICIjOTlDQ0ZGIiwgYWxwaGEgPSAwLjUpICsKICAjIEFkZCBhIHBvaW50IHdpdGggdGhlIG1lYW4gdmFsdWUKICBnZ3Bsb3QyOjpzdGF0X3N1bW1hcnkoCiAgICBnZW9tID0gInBvaW50IiwKICAgIGZ1biA9ICJtZWFuIiwKICAgICMgQ2hhbmdlIHRoZSBhZXN0aGV0aWNzIG9mIHRoZSBwb2ludHMKICAgIHNpemUgPSAzLAogICAgY29sb3IgPSAiIzAwNjZDQyIsCiAgICBzaGFwZSA9IDE4CiAgKSArCiAgIyBGbGlwIHRoZSBheGVzCiAgZ2dwbG90Mjo6Y29vcmRfZmxpcCgpICsKICBnZ3Bsb3QyOjpsYWJzKHRpdGxlID0gIlJhbmRvbSBHZW5lIFNldCBHU1ZBIFNjb3JlcyIsCiAgICAgICAgICAgICAgICB4ID0gImdlbmUgc2V0IHNpemUiLAogICAgICAgICAgICAgICAgeSA9ICJHU1ZBIHNjb3JlIikgKwogIGdncGxvdDI6OnRoZW1lX2J3KCkKYGBgCgpXaGF0IGRvIHlvdSBub3RpY2UgYWJvdXQgdGhlc2UgZGlzdHJpYnV0aW9ucz8KSG93IG1pZ2h0IHlvdSB1c2UgdGhpcyBpbmZvcm1hdGlvbiB0byBpbmZvcm0geW91ciBpbnRlcnByZXRhdGlvbiBvZiBHU1ZBIHNjb3Jlcz8KCiMjIyBIb3cgY2FuIHlvdSB1c2UgdGhlc2Ugc2NvcmVzPwoKSWYgeW91IGRpZCBoYXZlIGdyb3VwcyBpbmZvcm1hdGlvbiBmb3IgeW91ciBzYW1wbGVzLCB5b3UgY291bGQgdGVzdCBmb3IgcGF0aHdheSBzY29yZXMgZGlmZmVyZW5jZXMgYmV0d2VlbiBncm91cHMuCkhlcmUncyBbYW4gZXhhbXBsZV0oaHR0cHM6Ly9odG1scHJldmlldy5naXRodWIuaW8vP2h0dHBzOi8vZ2l0aHViLmNvbS9BbGV4c0xlbW9uYWRlL09wZW5QQlRBLWFuYWx5c2lzL2Jsb2IvOWI0NGJmMWMxODZiMzEyNmIxNmRiZTViODc3NTZiM2VhZTNmZWVjMi9hbmFseXNlcy9nZW5lLXNldC1lbnJpY2htZW50LWFuYWx5c2lzLzAyLW1vZGVsLWdzZWEubmIuaHRtbCkgb2YgdGhhdCBmcm9tIHRoZSBPcGVuUEJUQSBwcm9qZWN0IGl0c2VsZiEgCgpZb3UgY2FuIGFsc28gdmlzdWFsaXplIHRoaXMgbWF0cml4IGluIGEgaGVhdG1hcC4KSGVyZSdzIGEgZmlndXJlIGZyb20gdGhlIE9wZW5QQlRBIHByb2plY3QsIHdoZXJlIHRoZSBtaWRkbGUgcGFuZWwgaXMgYSBoZWF0bWFwIG9mIEdTVkEgc2NvcmVzIHRoYXQgd2VyZSBzaWduaWZpY2FudGx5IGRpZmZlcmVudCBiZXR3ZWVuIGhpc3RvbG9naWVzLgoKIVtdKGh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9BbGV4c0xlbW9uYWRlL09wZW5QQlRBLWFuYWx5c2lzLzliNDRiZjFjMTg2YjMxMjZiMTZkYmU1Yjg3NzU2YjNlYWUzZmVlYzIvZmlndXJlcy9wbmdzL3RyYW5zY3JpcHRvbWljLW92ZXJ2aWV3LnBuZykKClsqQ29kZSBoZXJlLipdKGh0dHBzOi8vZ2l0aHViLmNvbS9BbGV4c0xlbW9uYWRlL09wZW5QQlRBLWFuYWx5c2lzL2Jsb2IvOWI0NGJmMWMxODZiMzEyNmIxNmRiZTViODc3NTZiM2VhZTNmZWVjMi9maWd1cmVzL3NjcmlwdHMvdHJhbnNjcmlwdG9taWMtb3ZlcnZpZXcuUiNMMTA2KQoKIyMjIFdyaXRlIHJlc3VsdHMgdG8gZmlsZQoKYGBge3Igd3JpdGVfZ3N2YV9yZXN1bHRzfQpnc3ZhX3Jlc3VsdHMgJT4lCiAgYXMuZGF0YS5mcmFtZSgpICU+JQogIHRpYmJsZTo6cm93bmFtZXNfdG9fY29sdW1uKCJwYXRod2F5IikgJT4lCiAgcmVhZHI6OndyaXRlX3RzdihmaWxlID0gZ3N2YV9yZXN1bHRzX2ZpbGUpCmBgYAoKIyMgU2Vzc2lvbiBJbmZvCgpgYGB7ciBzZXNzaW9uX2luZm99CnNlc3Npb25JbmZvKCkKYGBgCg==